2 links tagged with all of: quantization + speculative decoding
Click any tag below to further narrow down your results
Links
Liquid AI released Pipette, an open-source benchmarking suite for measuring AI model performance on phones, laptops, and embedded devices, plus DSpark draft models that speed up inference by 2-3x through speculative decoding. Both tools come with public datasets and are designed to let developers test the exact hardware-model-runtime combinations they plan to ship.
- Pipette provides a standardized way to benchmark 35+ model classes across different quantization levels, runtimes, and devices (phones, laptops, PCs, embedded hardware), with 10k+ verified results already in the warehouse and an interactive dashboard to compare configurations.
- DSpark draft models achieve 2.27x to 3.18x throughput gains on everything from M4 Max MacBooks to H100 GPUs by using lightweight draft models to propose token blocks that a target model verifies in one pass, without changing output quality.
- Both projects are open-source with native iOS and Android apps, public datasets, and submission workflows that let developers contribute benchmarks for their own hardware and model combinations.
Meta open-sourced Muse Glimmer, a 30-billion-parameter model designed to run on consumer GPUs and laptops for local AI agent tasks like scheduling, coding, and tool use. The model uses quantization and speculative decoding to fit within 20-32GB of memory while maintaining speed for real-time interaction. It's available now under Apache 2.0 license with integrations for llama.cpp, MLX, and other frameworks.
- Meta open-sourced Muse Glimmer, a 30B-parameter agentic model quantized to under 20GB so it runs locally on consumer GPUs/laptops (24-32GB) while competing with Gemma 4-31B and Qwen 3.6-27B.
- Speculative decoding with a lightweight drafter model proposes token blocks at once instead of one token at a time, speeding up reasoning and tool calls without changing outputs.
- Training combined logit distillation from a larger teacher model, agent-heavy mid-training data, and post-training RL/distillation across reasoning, coding, and agentic tasks.
- Released under Apache 2.0 with upcoming llama.cpp, MLX, and ExecuTorch integrations and support from Ollama, LM Studio, Together AI, and chipmakers like NVIDIA, Intel, AMD, and Arm.